{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7a5d459f",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/uptrain-ai/uptrain/blob/main/examples/checks/conversation/number_of_turns.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f74cbf3-07b5-40d8-86cd-6fe807d8f36e",
   "metadata": {
    "tags": []
   },
   "source": [
    "<h1 align=\"center\">\n",
    "  <a href=\"https://uptrain.ai\">\n",
    "    <img width=\"300\" src=\"https://user-images.githubusercontent.com/108270398/214240695-4f958b76-c993-4ddd-8de6-8668f4d0da84.png\" alt=\"uptrain\">\n",
    "  </a>\n",
    "</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fc0b549-d081-442e-87ce-1ed0657540f2",
   "metadata": {},
   "source": [
    "<h1 style=\"text-align: center;\">Finding out the Number of Turns in Conversations</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f92208d-e85c-4108-951d-4f54940ace43",
   "metadata": {
    "tags": []
   },
   "source": [
    "**What is Number of Turns?**: The number of turns refers to the number of times the user and the LLM interact with each other in a conversation. \n",
    "\n",
    "For example, in the conversation below, the number of turns is 3.\n",
    "\n",
    "> **User:** Hi, I want to book an appointment.\n",
    ">\n",
    "> **LLM:** Sure, what would you like to book the appointment for?\n",
    ">\n",
    "> **User:** I would like to book an appointment for a haircut.\n",
    ">\n",
    "> **LLM:** Great! When would you like to book the appointment?\n",
    ">\n",
    "> **User:** I would like to book the appointment for tomorrow at 3 PM.\n",
    ">\n",
    "> **LLM:** Your appointment is booked for tomorrow at 3 PM. Thank you!\n",
    "\n",
    "\n",
    "\n",
    "**Data schema**: The data schema required for this evaluation is as follows:\n",
    "\n",
    "| Column Name | Description |\n",
    "| ----------- | ----------- |\n",
    "| conversation | The conversation between the user and the LLM |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82c5b58f-bfb9-4c63-8b22-8ed05fa1fa4d",
   "metadata": {},
   "source": [
    " If you face any difficulties, need some help with using UpTrain or want to brainstorm on custom evaluations for your use-case, [speak to the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min).\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f936a36",
   "metadata": {},
   "source": [
    "## Step 1: Install UpTrain by running 'pip install uptrain'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37a5ab48",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: uptrain in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (0.7.0)\n",
      "Requirement already satisfied: pydantic in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.6.4)\n",
      "Requirement already satisfied: pydantic-settings in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.2.1)\n",
      "Requirement already satisfied: loguru in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.7.0)\n",
      "Requirement already satisfied: lazy-loader in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.3)\n",
      "Requirement already satisfied: networkx in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (3.1)\n",
      "Requirement already satisfied: polars>=0.18 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.20.13)\n",
      "Requirement already satisfied: pandas in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2.0.3)\n",
      "Requirement already satisfied: numpy>=1.23.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.26.3)\n",
      "Requirement already satisfied: httpx>=0.24.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.26.0)\n",
      "Requirement already satisfied: plotly>=5.0.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (5.15.0)\n",
      "Requirement already satisfied: aiolimiter>=1.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.1.0)\n",
      "Requirement already satisfied: openai>=1.6.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.6.1)\n",
      "Requirement already satisfied: fsspec in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (2023.6.0)\n",
      "Requirement already satisfied: litellm in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.34.25)\n",
      "Requirement already satisfied: pyyaml in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (6.0.1)\n",
      "Requirement already satisfied: json5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (0.9.24)\n",
      "Requirement already satisfied: python-docx in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.1.0)\n",
      "Requirement already satisfied: pymupdf in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.23.25)\n",
      "Requirement already satisfied: faiss-cpu in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from uptrain) (1.7.4)\n",
      "Requirement already satisfied: anyio in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (3.7.1)\n",
      "Requirement already satisfied: certifi in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (2023.7.22)\n",
      "Requirement already satisfied: httpcore==1.* in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (1.0.2)\n",
      "Requirement already satisfied: idna in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (3.4)\n",
      "Requirement already satisfied: sniffio in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (1.3.0)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.24.1->uptrain) (0.14.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (1.8.0)\n",
      "Requirement already satisfied: tqdm>4 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (4.66.2)\n",
      "Requirement already satisfied: typing-extensions<5,>=4.7 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (4.7.1)\n",
      "Requirement already satisfied: tenacity>=6.2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from plotly>=5.0.0->uptrain) (8.2.3)\n",
      "Requirement already satisfied: packaging in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from plotly>=5.0.0->uptrain) (23.2)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pydantic->uptrain) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.16.3 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pydantic->uptrain) (2.16.3)\n",
      "Requirement already satisfied: aiohttp in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (3.9.1)\n",
      "Requirement already satisfied: click in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (8.1.6)\n",
      "Requirement already satisfied: importlib-metadata>=6.8.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (6.8.0)\n",
      "Requirement already satisfied: jinja2<4.0.0,>=3.1.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (3.1.2)\n",
      "Requirement already satisfied: python-dotenv>=0.2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (1.0.0)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.31.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (2.31.0)\n",
      "Requirement already satisfied: tiktoken>=0.4.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (0.4.0)\n",
      "Requirement already satisfied: tokenizers in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from litellm->uptrain) (0.15.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2023.3)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pandas->uptrain) (2023.3)\n",
      "Requirement already satisfied: PyMuPDFb==1.23.22 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from pymupdf->uptrain) (1.23.22)\n",
      "Requirement already satisfied: lxml>=3.1.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from python-docx->uptrain) (4.9.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from importlib-metadata>=6.8.0->litellm->uptrain) (3.16.2)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from jinja2<4.0.0,>=3.1.2->litellm->uptrain) (2.1.3)\n",
      "Requirement already satisfied: six>=1.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->uptrain) (1.16.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm->uptrain) (3.2.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm->uptrain) (1.26.16)\n",
      "Requirement already satisfied: regex>=2022.1.18 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from tiktoken>=0.4.0->litellm->uptrain) (2023.8.8)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (23.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.9.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.3.1)\n",
      "Requirement already satisfied: huggingface_hub<1.0,>=0.16.4 in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from tokenizers->litellm->uptrain) (0.20.3)\n",
      "Requirement already satisfied: filelock in /Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers->litellm->uptrain) (3.12.2)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install uptrain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb91700c",
   "metadata": {},
   "source": [
    "## Step 2: Let's define our dataset to run evaluations upon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bf4c2b81-59f4-4faa-b474-7b3f4c2ed509",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "short_conversation = [{\n",
    "    'conversation' : [\n",
    "        {\"role\": \"patient\", \"content\": \"Help\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"what do you need\"}, \n",
    "        {\"role\": \"patient\", \"content\": \"Having chest pain\"}, \n",
    "        {\"role\": \"nurse\", \"content\": \"please call 102\"},\n",
    "        {\"role\": \"patient\", \"content\": \"Thank you nurse\"}, \n",
    "    ]  \n",
    "}]\n",
    "\n",
    "long_conversation = [{\n",
    "    'conversation' : [\n",
    "        {\"role\": \"patient\", \"content\": \"Hello\"},\n",
    "        {\"role\": \"nurse\", \"content\": \"Hello, how can I help you\"},\n",
    "        {\"role\": \"patient\", \"content\": \"I am feeling very sick\"},\n",
    "        {\"role\": \"nurse\", \"content\": \"What are your symptoms\"},\n",
    "        {\"role\": \"patient\", \"content\": \"I have a headache\"},\n",
    "        {\"role\": \"nurse\", \"content\": \"Have you taken any medication\"},\n",
    "        {\"role\": \"patient\", \"content\": \"No, I haven't\"},\n",
    "        {\"role\": \"nurse\", \"content\": \"Please take some paracetamol\"},\n",
    "        {\"role\": \"patient\", \"content\": \"Thank you nurse\"},\n",
    "    ]  \n",
    "}]\n",
    "\n",
    "data = short_conversation + long_conversation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b640784e",
   "metadata": {},
   "source": [
    "## Step 3: Running evaluations using UpTrain's Open-Source Software (OSS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d6706deb-8e9d-4d45-8f9b-aed0e9da608f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dhruvchawla/work/uptrain-v1/.venv/lib/python3.11/site-packages/lazy_loader/__init__.py:185: RuntimeWarning: subpackages can technically be lazily loaded, but it causes the package to be eagerly loaded even if it is already lazily loaded.So, you probably shouldn't use subpackages with this lazy feature.\n",
      "  warnings.warn(msg, RuntimeWarning)\n",
      "\u001b[32m2024-05-17 16:07:33.382\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36muptrain.operators.language.llm\u001b[0m:\u001b[36mfetch_responses\u001b[0m:\u001b[36m268\u001b[0m - \u001b[33m\u001b[1mDetected a running event loop, scheduling requests in a separate thread.\u001b[0m\n",
      "100%|██████████| 2/2 [00:02<00:00,  1.37s/it]\n",
      "/Users/dhruvchawla/work/uptrain-v1/uptrain/operators/language/llm.py:271: RuntimeWarning: coroutine 'LLMMulticlient.async_fetch_responses' was never awaited\n",
      "  with ThreadPoolExecutor(max_workers=1) as executor:\n",
      "RuntimeWarning: Enable tracemalloc to get the object allocation traceback\n"
     ]
    }
   ],
   "source": [
    "from uptrain import EvalLLM, ConversationNumberOfTurns\n",
    "import json\n",
    "\n",
    "OPENAI_API_KEY = \"sk-***************************\"  # Insert your OpenAI key here\n",
    "\n",
    "eval_llm = EvalLLM(openai_api_key=OPENAI_API_KEY)\n",
    "\n",
    "res = eval_llm.evaluate(\n",
    "    data = data,\n",
    "    checks = [ConversationNumberOfTurns(user_persona=\"patient\", llm_persona=\"nurse\")],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fa5df117-5516-4e32-b86b-e3ade08a5d78",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"please call 102\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_conversation_number_of_turns\": 3,\n",
      "      \"explanation_conversation_number_of_turns\": {\n",
      "         \"Reasoning\": [\n",
      "            \"The user starts by asking for help, which is responded to by the nurse asking what is needed. This is the first turn.\",\n",
      "            \"The user then mentions having chest pain, to which the nurse responds by advising to call 102. This is the second turn.\",\n",
      "            \"Finally, the user thanks the nurse, concluding the conversation. This is the third turn.\"\n",
      "         ],\n",
      "         \"Turns\": 3\n",
      "      },\n",
      "      \"conversation_length\": 5\n",
      "   },\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Hello\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Hello, how can I help you\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"I am feeling very sick\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"What are your symptoms\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"I have a headache\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Have you taken any medication\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"No, I haven't\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Please take some paracetamol\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_conversation_number_of_turns\": 5,\n",
      "      \"explanation_conversation_number_of_turns\": {\n",
      "         \"Reasoning\": [\n",
      "            \"The conversation starts with the patient greeting the nurse.\",\n",
      "            \"The patient expresses feeling sick, prompting the nurse to ask about symptoms.\",\n",
      "            \"The patient mentions having a headache, leading the nurse to inquire about medication.\",\n",
      "            \"The patient confirms not taking any medication, and the nurse advises taking paracetamol.\",\n",
      "            \"The patient thanks the nurse, indicating the resolution of the user query.\",\n",
      "            \"Therefore, the number of turns in the conversation taken to resolve the user query is 5.\"\n",
      "         ],\n",
      "         \"Turns\": 5\n",
      "      },\n",
      "      \"conversation_length\": 9\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res,indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49ea84a4-2846-4a3c-97c2-ac8c8d413c52",
   "metadata": {},
   "source": [
    "## Step 4: Let's look at some of the results "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db249e6d-5a30-43aa-92f0-6217f6ca85f3",
   "metadata": {},
   "source": [
    "### Sample with a good conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6343e0c0-35a1-4d59-a771-dd8a0116fb50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "   \"conversation\": [\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Help\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"what do you need\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Having chest pain\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"please call 102\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Thank you nurse\"\n",
      "      }\n",
      "   ],\n",
      "   \"score_conversation_number_of_turns\": 3,\n",
      "   \"explanation_conversation_number_of_turns\": {\n",
      "      \"Reasoning\": [\n",
      "         \"The user starts by asking for help, which is responded to by the nurse asking what is needed. This is the first turn.\",\n",
      "         \"The user then mentions having chest pain, to which the nurse responds by advising to call 102. This is the second turn.\",\n",
      "         \"Finally, the user thanks the nurse, concluding the conversation. This is the third turn.\"\n",
      "      ],\n",
      "      \"Turns\": 3\n",
      "   },\n",
      "   \"conversation_length\": 5\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res[0],indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a7388f1",
   "metadata": {},
   "source": [
    "### Sample with a bad conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9c1a6497",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "   \"conversation\": [\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Hello\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"Hello, how can I help you\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"I am feeling very sick\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"What are your symptoms\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"I have a headache\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"Have you taken any medication\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"No, I haven't\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"nurse\",\n",
      "         \"content\": \"Please take some paracetamol\"\n",
      "      },\n",
      "      {\n",
      "         \"role\": \"patient\",\n",
      "         \"content\": \"Thank you nurse\"\n",
      "      }\n",
      "   ],\n",
      "   \"score_conversation_number_of_turns\": 5,\n",
      "   \"explanation_conversation_number_of_turns\": {\n",
      "      \"Reasoning\": [\n",
      "         \"The conversation starts with the patient greeting the nurse.\",\n",
      "         \"The patient expresses feeling sick, prompting the nurse to ask about symptoms.\",\n",
      "         \"The patient mentions having a headache, leading the nurse to inquire about medication.\",\n",
      "         \"The patient confirms not taking any medication, and the nurse advises taking paracetamol.\",\n",
      "         \"The patient thanks the nurse, indicating the resolution of the user query.\",\n",
      "         \"Therefore, the number of turns in the conversation taken to resolve the user query is 5.\"\n",
      "      ],\n",
      "      \"Turns\": 5\n",
      "   },\n",
      "   \"conversation_length\": 9\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res[1],indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2904e2f4",
   "metadata": {},
   "source": [
    "## [Optional] Step 5: UpTrain Managed Service and Dashboards"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bb801c3",
   "metadata": {},
   "source": [
    "You can create a free UpTrain account [here](https://uptrain.ai/) and get free trial credits. If you want more trial credits, [book a call with the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5e23c4a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-05-17 16:07:36.302\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.remote\u001b[0m:\u001b[36mlog_and_evaluate\u001b[0m:\u001b[36m677\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain server\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"please call 102\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_conversation_number_of_turns\": 3,\n",
      "      \"explanation_conversation_number_of_turns\": \"The user query was resolved in 3 turns. The patient asked for help, mentioned having chest pain, and the nurse instructed to call 102, which resolved the query.\",\n",
      "      \"conversation_length\": 5\n",
      "   },\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Hello\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Hello, how can I help you\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"I am feeling very sick\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"What are your symptoms\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"I have a headache\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Have you taken any medication\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"No, I haven't\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Please take some paracetamol\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_conversation_number_of_turns\": 6,\n",
      "      \"explanation_conversation_number_of_turns\": \"The conversation consists of 6 turns to resolve the user query. Each turn includes an exchange between the patient and the nurse, starting from the initial greeting to the resolution of the patient's query about feeling sick and receiving advice to take paracetamol.\",\n",
      "      \"conversation_length\": 9\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "from uptrain import APIClient, Settings, ConversationNumberOfTurns\n",
    "\n",
    "UPTRAIN_API_KEY = \"up-***************************\"  # Insert your UpTrain API key here\n",
    "\n",
    "uptrain_client = APIClient(\n",
    "    Settings(\n",
    "        uptrain_access_token=UPTRAIN_API_KEY,\n",
    "    )\n",
    ")\n",
    "\n",
    "res = uptrain_client.log_and_evaluate(\n",
    "    \"Emergency-Contact-Information\",\n",
    "    data=data, \n",
    "    checks=[ConversationNumberOfTurns(user_persona=\"patient\", llm_persona=\"nurse\")],\n",
    ")\n",
    "\n",
    "print(json.dumps(res, indent=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f478f4f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-05-17 16:07:36.302\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.remote\u001b[0m:\u001b[36mlog_and_evaluate\u001b[0m:\u001b[36m677\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain server\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Help\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"what do you need\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Having chest pain\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"please call 102\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_conversation_number_of_turns\": 3,\n",
      "      \"explanation_conversation_number_of_turns\": \"The user query was resolved in 3 turns. The patient asked for help, mentioned having chest pain, and the nurse instructed to call 102, which resolved the query.\",\n",
      "      \"conversation_length\": 5\n",
      "   },\n",
      "   {\n",
      "      \"conversation\": [\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Hello\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Hello, how can I help you\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"I am feeling very sick\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"What are your symptoms\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"I have a headache\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Have you taken any medication\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"No, I haven't\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"nurse\",\n",
      "            \"content\": \"Please take some paracetamol\"\n",
      "         },\n",
      "         {\n",
      "            \"role\": \"patient\",\n",
      "            \"content\": \"Thank you nurse\"\n",
      "         }\n",
      "      ],\n",
      "      \"score_conversation_number_of_turns\": 6,\n",
      "      \"explanation_conversation_number_of_turns\": \"The conversation consists of 6 turns to resolve the user query. Each turn includes an exchange between the patient and the nurse, starting from the initial greeting to the resolution of the patient's query about feeling sick and receiving advice to take paracetamol.\",\n",
      "      \"conversation_length\": 9\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "from uptrain import APIClient, Settings, ConversationNumberOfTurns\n",
    "\n",
    "UPTRAIN_API_KEY = \"up-***************************\"  # Insert your UpTrain API key here\n",
    "\n",
    "uptrain_client = APIClient(\n",
    "    Settings(\n",
    "        uptrain_access_token=UPTRAIN_API_KEY,\n",
    "    )\n",
    ")\n",
    "\n",
    "res = uptrain_client.log_and_evaluate(\n",
    "    \"Emergency-Contact-Information\",\n",
    "    data=data, \n",
    "    checks=[ConversationNumberOfTurns(user_persona=\"patient\", llm_persona=\"nurse\")],\n",
    ")\n",
    "\n",
    "print(json.dumps(res, indent=3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
